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1.  Introduction 


Phase  II  of  this  project  shifted  the  focus  towards  the  design  and  development  of  3D  and 
display/presentation  technology  to  ensure  a  more  realistic  mobile  emergency  simulation 
application  is  presented  to  users.  A  “game  engine”  was  introduced  to  merge  different  display 
and  simulation  technologies  into  one  seamless  system  and  tested  in  a  real  life  situation. 

>  Design  and  develop  3D  and  display/presentation  technology  that  will  ensure  a 
more  realistic  simulation  is  presented  to  users. 

>  Evaluate  how  different  display  and  simulation  technologies  work  in  real  life 
situations. 


2.  Body 

The  need  to  plan  and  calculate  the  effects  of  major  emergencies  including  terrorist’s  attacks 
has  become  critical  in  the  US  Army.  To  ensure  optimal  military  readiness,  there  is  an  urgent 
need  to  realistically  simulate  these  events.  Development  of  a  realistic  interactive  emergency 
simulation  and  response  system  will  help  to  ensure  that  US  Army  management  and 
emergency  personnel  are  adequate  trained  and  prepared  to  respond  quickly  and  decisively  to 
any  act  of  terrorism. 

The  US  Army  and  society  in  general  is  rapidly  developing  towards  greater  dependence  on 
fragile  technological  and  management  systems  vulnerable  to  natural  disasters  and  terror 
assaults. 

There  is  a  need  in  the  US  Army  to  simulate  crisis  situations.  These  simulations  can  be  used  to 
calculate  the  effects  of  different  actions  and  countermeasures.  Most  importantly,  these 
simulations  can  be  used  to  train  US  Army  management  and  emergency  personnel. 

•  The  US  Government  (Department  of  Homeland  Security)  has  prepared  a 

set  of  master  scenarios  in  the  event  of  emergencies  or  hostile  strikes  on  US  soil.  To 
ensure  civil  and  military  readiness,  these  and  other  scenarios  are  regularly  simulated 

•  The  quality  and  learning  from  these  simulations  can  be  enhanced.  There  are  also 
critical  and  prioritized  issues  in  actual  emergency  and  crisis  management,  strategic  as 
well  as  a  tactical,  to  further  narrow  the  gap  between  simulation  and  the  real  situation 

•  Valuable  information  on  Homeland  Security  needs  to  be  simply  and  efficiently 
communicated  between  the  260  million  American  mobile  subscribers  and  the  relevant 
authorities 

Mobile  Media  Inventions  AS  has  developed  a  set  of  technologies  that  when  merged  together 
can  produce  the  framework  for  this  type  of  simulation  system.  These  technologies  have 
evolved  from  medical  and  entertainment  3D  animation,  mobile  telecom  interactive  technology 
and  satellite  and  aerial  photography.  When  merged,  developed,  backed  up  by  sufficient 
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statistical  data  and  installed  in  a  main  frame  computer  system,  a  structural  and  sociological 
model  can  be  generated. 

The  project  has  found  a  solution  for  a  set  of  issues  related  to  geographical  database 
technology,  automatic  interpretation  of  high-resolution  satellite  images  and  stereo  models  and 
representation  and  rendering  on  different  devices. 

The  concept  comprises  extensive  use  of  high-resolution  optical  imagery.  Generation  of  stereo 
models  consists  of  determining  the  acquisition  geometry  of  images  that  will  be  needed  to 
retrieve  the  3D  positions  of  details  of  the  ground  surface  and  its  objects.  In  order  to  generate 
the  stereo  model,  a  model  is  needed  for  image  acquisition  geometry  and  we  need  to  determine 
the  parameters  of  the  model.  These  are  determined  by  means  of  corresponding  tie  points 
between  the  images  and  ground  control  points,  which  refer  to  a  global  co-ordinate  system. 

The  introduction  of  Java  and  picture  phones  to  the  marketplace  has  opened  totally  new 
possibilities  within  this  field.  Utilizing  these  technologies  makes  it  possible  to  integrate 
mobile  devices/users  by  collecting  pictures  from  the  user  and  sending  pictures/maps  and  other 
information  to  them.  It  is  also  possible  to  distribute  applications  to  be  used  for  a  particular 
event,  in  real-time. 

These  applications  has  been  tailored  to  guide  the  user  in  an  emergency  situation  and/or 
retrieve  fundamental  multimedia  information  from  the  scene  and  relayed  back  to  the 
emergency  center.  This  could  be  a  medical  person  who  is  not  at  his  desk  or  in  the  simulation 
center.  He  will  take  part  in  the  simulation  by  using  his  mobile  phone  by  receiving  information 
(pictures,  text,  video)  for  the  simulator  and  be  able  to  play  a  scenario  at  any  given  time 
(without  notice)  and  involve  the  resources  needed  to  solve  the  specific  situation. 

In  the  situation,  the  phone  will  be  used  by  real  people  involved  in  the  simulation  to  report 
status  (text/pictures/video)  back  to  the  simulator.  These  data  is  integrated  in  which  the  objects 
in  the  simulation  model.  It  can  also  be  used  to  track  movements  of  real  objects  in  a  real 
situation.  In  this  case,  the  Middletown200I0  simulator  is  more  a  information  system.  In  this 
manner,  real  situation  data  can  be  transferred  and  merged  into  the  3D  simulation  model  and 
visualized. 

From  a  US  Army  stand  point,  defensive  rescue  scenarios  and  situations  could  be  simulated 
parallel  to  offensive  interventions  and  operations.  The  simulator  will  also  allow  for  very 
specific  urban  warfare  scenarios.  These  simulations  would  create  possibilities  to  test  different 
tactical  options  in  a  given  situation  from  an  offensive,  as  well  as,  defensive  point  of  view. 
Realism  is  a  key  word  in  connection  with  the  concept. 

Mobile  device  technology 

Mobile  devices  have  been  dramatically  more  powerful  the  last  years  and  equipped  with 
advanced  functionality  like  GPS,  camera  and  video.  Today’s  mobile  devices  are  more 
powerful  than  PC’s  were  some  years  ago  and  the  graphic  quality  will  increase  even  more  in 
the  next  years  with  the  introduction  of  new  graphic  processors.  Demos  developed  in  Phase  II 
have  shown  the  important  role  that  mobile  devices  have  as  both  display  and  information 
gathering  units  in  the  future.  However,  as  the  size  of  the  devices  will  stay  the  same  due  to 
usability  reasons,  it  is  important  to  design  applications  with  very  good  user  interfaces  and 
simple,  but  advanced  functionality. 
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In  advanced  simulation  and  response  applications,  such  as  the  Middletown  system,  multiple 
viewers  are  using  different,  personal  mobile  devices  to  drill  into  the  same  information. 

Emergencies  and  Crisis  situations  can  be  reduced  in  severity  or  avoided  all  together  with 
better  information  systems.  For  example,  when  Katrina  hit  the  USA,  the  simple  fact  that  it 
was  difficult  to  locate  family  and  friends  made  rescue  work  much  more  difficult  on  many 
levels,  in  addition  to  being  costly  in  terms  of  human  life  as  well  as  expensive.  We  have 
created  a  system  that  in  an  intelligent  way  can  use  a  person’s  cell  phone  as  a  critical  device  to 
receive  and  give  critical  information.  The  solution  is  suitable  for  both  Government  Agency 
and  private  use. 

Figure  1.  The  Middletown  client  network  and  user  interface 


"And  I'm  informing  family  and  friends" 


The  Middletown  Client  is  an  application  to  be  installed  on  users’  mobile  devices.  It 
communicates  via  internet  to  the  Middletown  Server.  The  Middletown  Client  will 
integrate  with  third-party  content  providers  within  the  area  of  Homeland  Security  and 
Emergency  Services.  The  Middletown  Client  contains  the  telephone  numbers  and 
addresses  of  the  Fire  Department,  Police,  Ambulance,  Middletown  Control  Center  and 
other  important  Emergency  Services.  The  user  will  be  able  to  immediately  contact  these 
Emergency  units  via  the  simple  push  of  a  button. 

The  solution  comprises  extensive  use  of  high-resolution  optical  imagery.  The  generation 
of  stereo  models  consists  of  determining  the  acquisition  geometry  of  images  that  will  be 
needed  to  retrieve  the  3D  positions  of  details  of  the  ground  surface  and  its  objects.  In 
order  to  generate  the  stereo  model,  a  model  is  needed  for  image  acquisition  geometry  and 
we  need  to  determine  the  parameters  of  it.  These  are  determined  by  means  of 
corresponding  tie  points  between  the  images  and  ground  control  points,  which  refer  to  a 
global  co-ordinate  system. 

The  Middletown  can  provide  an  advanced  simulation  solution  to  achieve  a  better  training 
ground.  It  will  be  delivered  in  the  shape  of  a  scalable  client/server  solution  consisting  of 
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hardware,  software  and  interfaee  eonneetors  to  other  infrastruetures,  sueh  as  mobile 
networks.  The  main  produet  is  a  server-elient.  A  Java  elient  will  be  talking  with  this  elient 
from  a  mobile  deviee.  The  serviee  will  provide  position  based  in  real  time.  Thus  both  time 
and  plaee  will  be  used  to  target  speeifie  information  relevant  for  the  partieular  users.  Then 
information  will  eome  from  the  web  and  typieally  be  related  to  seeurity.  An  individual  on 
the  move  in  any  terrain  will  be  offered  different  seeurity  and  erisis-  related  information 
that  meets  his  or  her  needs  as  time,  and  their  position  ehange.  Ease  of  use  and  effieieney 
of  information  delivery  eombine  to  make  this  a  highly  advaneed  applieation  in  whieh  time 
and  position  are  eritieal  faetors. 


Figure  2.  Middletown  mobile  user  interface 


The  mobile  user  ean  aeeess  several  levels  of  information.  The  First  menu  level  ean  give 
easy  aeeess  to  funetions  sueh  as: 

1 .  Medieal  emergeneies 

2.  Traffie  situations  and  status 

3.  Explosions 

4.  Weather 

5.  Bioehemieal  information 

In  the  ease  of  a  bioehemieal  emergeney,  the  Seeond  layer  ean  give  biomedieal  information 
listing  different  eompounds  eonsidered  dangerous.  The  Third  level  will  show  a  situation 
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or  location  of  interest.  Note  that  all  this  information  will  change  according  to  accreditation 
of  information,  interest,  position  and  time.  The  interactivity  of  dynamic  information  gives 
a  unique  solution  system. 

Figure  3.  The  Middletown  Technical  infrastructure  solution 


Guardian 

Angel 


GPS 


Specifications 

1.  Instructions 

Broadcasting  of  vital  information  on  how  to  behave  in  a  crisis  situation. 

2.  Guardian  Angel 

Putting  the  user  in  touch  with  a  contact  person  or  “Guardian  Angel”, 
who  can  give  medical  or  other  assistance  remotely. 

3.  Family,  Neighbors  &  Friends 

Communication  between  the  user  and  the  control  center  enable  both  to  receive  and 
deliver  quick  and  reliable  information  to  family  members,  neighbors  or  friends. 

3.  Localization 

Positioning  so  that  relevant  information  can  be  exchanged  between  the  mobile  user  and 
Middletown. 
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Figure  4.  Middletown  interface  with  Google  Earth 


Google  Earth  has  earned  tremendous  interest,  however  this  is  as  far  as  it  goes. . .. 


This  illustrates  the  Middletown  eoneept:  we  ean  traek  mobile  users  in  a  building  and  in  a  3D 
model.  Also,  the  mobile  eell  phone  owners  ean  eommunieate  to  the  eentral  with  real  time 
voiee,  sms,  video  and  pietures.  Coupled  with  aeeessing  floor  plans,  3D  models  of  buildings 
from  third  party  providers  within  minutes  a  broad  set  of  eritieal  seenarios  ean  be  better  dealt 
with. 
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3.  Key  Research  Accomplishments 


A  prototype  has  been  developed  to  handle  positioned  based  photos,  videos,  voice  and 
environmental  data.  This  prototype  forms  the  basis  for  the  following  solutions: 

•  Middletown  can  provide  an  advanced  simulation  solution  to  provide  better  training.  It 
can  be  delivered  in  the  shape  of  a  scalable  client/server  solution  consisting  of 
hardware,  software  and  interface  connectors  to  other  infrastructures,  such  as  mobile 
networks. 

•  The  Army  and  other  designated  users  will  be  able  to  interact  with  both  the  simulator 
and  real  people  connecting  to  the  system  through  internet  and  mobile  phone 
infrastructure. 

•  In  crises,  the  simulator  can  transform  from  a  training  tool  to  an  information  and  “what- 
if  ’  planning  system.  The  simulated  objects  will  then  be  replaced  by  real  objects  which 
are  controlled  by  real  events. 

•  Information  regarding  these  events  is  fed  into  the  system  from  various  sources  such  as 
mobile  phones  (positioning,  pictures,  text,  videos)  and  input  through  the  systems 
graphical  user  interface. 


4.  Reportable  Outcomes 

Publications:  None 

Preparation  for  an  application  to  create  a  fully  operational  version  based  on  research  findings 
in  Phase  I  and  II  is  in  progress.  Additional  funding  is  being  explored  both  in  the  US  and  in 
Norway. 

5.  Conclusion 

Middletown  2010  project  has  progressed  further  in  Phase  II  to  achieve  the  goal  of  building  the 
defined  simulator.  These  are: 

•  Automatic  generation  of  urban  scenery  models  from  satellite  images  and  mobile 
photos 

•  User  interfaces  to  tackle  fast  and  reliable  position  based  interaction  with  mobile 
devices 

In  Phase  II  we  have  worked  on  how  to  create  high  quality  models  of  urban  places  (houses, 
buildings,  etc.)  based  on  air-  and  satellite  photos.  More  work  is  need  here  to  allow  a  semi¬ 
automatic  3D  display  on  a  mobile  phone  based  on  a  series  of  2D  photographs  taken  either  by 
public  at  large  or  emergency  personnel. 

Before  a  fully  operational  version  of  Middletown  2010  is  ready,  there  is  a  need  to  address  the 
integration  issue  with  other  technical  platforms  more  in  depth.  This  has  been  addressed  from 
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the  start  but  the  development  towards  more  open  and  standard  solutions  for  data  and  serviee 
interaetion  suggests  resourees  ean  be  saved  by  integrating  Middletown  with  existing 
solutions.  Our  plan  eontinues  to  build  upon  what  is  already  invested  in  other  projeets. 


The  issue  of  user  interfaee  (how  the  user  interaets  with  the  Middletown  system)  has  been 
addressed  in  more  detail  during  phase  IT  We  have  developed  a  mobile  applieation  and  worked 
to  find  solutions  that  support  the  researeh  of  building  urban  3D  models  faster  and  with  higher 
quality.  The  result  of  Phase  II  indieates  that  it  is  possible  to  take  images  from  Google  earth 
and  similar  publiely  known  sourees  and  integrate  individual  positions  based  photographs  from 
the  publie  or  emergeney  personnel  to  eapture  effeetively  what  is  going  on  the  ground. 
Together,  these  two  image  sourees  ean  form  the  basis  for  a  fully  operational  Middletown 
elient-server  applieation. 

6.  References 

None 

7.  Appendices 

None 


12 


